Metrology applied to performance analysis using hardware counters

Performance evaluation and analysis are used to understand and improve computer system performance. One common approach is to measure performance indicators of interest during execution of a set of benchmark programs on a given system. Processor architecture, compiler, operating system, etc. and their interaction are often not predictable and can introduce random variability in such measurements. For instance, complex out-of-order processors with speculation may have a different memory utilization, instructions per cycle, pipeline stalls, branch prediction results, for each execution of a program. This raises the question of accuracy in collecting execution-based measurements. Modern microprocessors contain hardware performance counters to make such measurements and to collect dynamic information on the state of a processor [13], [15]. A common practice is to collect multiple measurements per counter and to report the average value and the standard deviation. However, only a limited number of hardware counters can be collect in one execution of a program. Therefore, the number of times a benchmark suite needs to be executed to collect all the counters of interest may be quite large. The question then is, what is the minimum number of samples required required to measure all parameters with a required accuracy? Another question is if measurements performed in separate benchmark executions can be combined. Finally, there is an issue of how to deal with measurement errors in systems that can collect only a small number of counters per run. The are the issues addressed in this work. We propose a methodology for hardware counter based measurements that answers the above